Papers with non-linguistic training
Learning Non-linguistic Skills without Sacrificing Linguistic Proficiency (2023.acl-long)
Copied to clipboard
| Challenge: | Numeracy is the most prevalent form of non-linguistic information embedded in textual corpora. |
| Approach: | They propose a framework for non-linguistic skill injection for LLMs that incorporates information-theoretic interventions and skill-specific losses to enable the learning of strict arithmetic reasoning. |
| Outcome: | The proposed model outperforms the state-of-the-art on injected non-linguistic skills and on linguistic knowledge retention with a fraction of the non-language training data (1/4) and zero additional synthetic linguistic training data. |